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Study Methods 


For the report: 

Opportunities and Obstacles: 

Implementing Stimulus-Funded School Improvement Grants in 
Maryland, Michigan, and Idaho 


The purpose of this study was to describe how Title I 1003(g) School Improvement 
Grants awarded through the American Reinvestment and Recovery Act (ARRA) in the 
2010-11 school year were implemented at the state, district, and local levels in three 
specific states (Idaho, Maryland, and Michigan). The study also examined participants’ 
views of the successes and challenges of implementation. 

Participants 

Participants for this study were selected using a number of purposeful sampling 
strategies. Through purposeful sampling, we hoped to make the best use of our resources 
by selecting “information-rich cases. . . from which one can learn a great deal about issues 
of central importance to the purpose of the inquiry” (Patton, 2002, p. 230). Though a 
small sample size prevented us from making empirical generalizations (which, as noted, 
is not the purpose of our study), purposeful sampling did, as Patton explains, “yield 
insights and in-depth understandings” about the ARRA SIG program within each case (p. 
230). 

Selection of states 

The three state participants (each serving as a separate case study) were selected for 
maximum variety sampling (Patton, 2002) based on the criteria of geographic location, 


the types of schools in each state receiving ARRA SIGs, and the school improvement 
models chosen by those schools in each state. 


For example: 

• In Idaho, our Western state, ARRA SIG tier 1 and 2 1 schools (N=13) are 
predominately rural, and those receiving grants (N=6) are using the 
transformation model. 

• In Maryland, our mid- Atlantic state, ARRA SIG tier 1 and 2 schools (N=16) are 
mostly urban. Schools receiving grants (N=l 1) are in Baltimore or Prince 
George’s County, a suburb of Washington, D.C., and are using the restart and 
turnaround models. 

• In Michigan, our Midwestern state, ARRA SIG tier 1 and 2 schools (N=108) are 
spread throughout the state in mostly urban and suburban areas, with a just few 
rural schools represented. Schools receiving grants (N=28) are using the 
turnaround and transformation models. 

Selection of ARRA SIG schools 

CEP selected two schools receiving ARRA SIG funds in each state for the study, 
following Patton’s (2002) maximum variety sampling. CEP researchers consulted with 
state education agency personnel and reviewed ARRA SIG data in Hurlburt et al., 2011 
to select a sample of schools in each state that represented a variety of ARRA SIG reform 
models and school locales (i.e. rural, urban, suburban). For example: 


'Under revised federal guidance, states must identify three tiers of schools eligible for ARRA SIGs. Tiers 1 
and 2 consist of the “persistently lowest-achieving” schools, which are among the lowest-achieving 5% in 
the state and receive highest priority for funding. Tier 3, which consists of other schools identified for 
improvement after failing to meet the accountability requirements of the No Child Left Behind Act 
(NCLB), may be considered for funding only after schools in tiers 1 and 2 have been funded. More 
information about these tiers is included in the full report. 


• In Michigan, two-thirds of the ARRA SIG recipient schools in the state are 
implementing the transformation model, while one-third are implementing the 
turnaround model. These schools are predominantly urban and suburban. In 
Michigan, we selected one urban school in a large city that was implementing the 
turnaround model and one urban school in a small city that was implementing the 
transformation model. 

• In Maryland, the state has 7 urban (Baltimore) and 4 suburban (Prince George’s 
County) schools. Five of these schools are implementing the restart model, and 6 
are implementing the turnaround model. Therefore, we selected one school in 
Baltimore that is implementing the restart model and one school in Prince 
George’s County that is implementing the turnaround model. 

• In Idaho, 5 of 6 eligible schools are rural, and the remaining school is suburban. 
All of these schools chose the transformation model. Therefore, we chose one 
rural and one suburban school in Idaho, both implementing the transformation 
model. 

In Maryland, researchers also consulted with district-level personnel about school 
selection. The schools selected for this study represent key elements of ARRA SIG as 
implemented in each state but are not necessarily representative of the entire population 
of schools within that state. 

Selection of comparison schools 

For each of the two schools receiving ARRA SIG funds selected in each state, CEP 
attempted to select a matching school (ideally in the same district) that was identified as 
tier 1 or 2 under ARRA SIG but was not awarded these funds in 2010-1 1. These non- 
recipient schools, when possible, were selected based on homogenous sampling, which 
involved matching the two schools selected that are receiving ARRA SIG funds with 
non-recipient schools that are similar based on criteria such as grade levels served, locale, 



geographic location, percentage of non-white students, percentage of low-income 
students, and percentage of non-proficient students on state tests. 


In Michigan, we were able to follow our criteria as described above. In Idaho, matching 
the rural recipient school with another non-recipient in the same district was not possible, 
since each district had only one tier 1 or 2 school. Therefore, after consultation with state 
education department personnel, CEP researchers matched the rural school in Idaho with 
a school in another district that was similar in size and the other criteria. The suburban 
school in Idaho could not be matched since there were no other suburban schools in tier 1 
or 2. 

In Maryland, the Baltimore City Public Schools had tier 1 and 2 schools that were 
eligible for but did not receive ARRA SIG funds, but these schools did not wish to 
participate in the study; therefore, we were not able to identify matching non-recipient 
schools for that district. The Prince George’s County Public Schools had no tier 1 and 2 
schools that did not receive ARRA SIG funds; therefore, we selected two schools in tier 3 
from this district. 

We believe that this matching system resulted in a sample of school pairs in each state 
that are similar on some demographic and contextual variables. However, we 
acknowledge that the schools do not represent a perfect comparison. As expected, 
interviews captured some, but not all of these differences; therefore, comparisons were 
made cautiously. 

Data Sources 

Data were collected for this study from state and local participants and from the public 
domain. More specifically, in order to explore our research questions, data were collected 
and analyzed from the following sources: 

• Interviews at the state, district, and school levels 

• State and local documents pertaining to ARRA SIG 



• An existing U.S. Department of Education data set containing demographic 
information, funding amounts, and ARRA SIG model selection information for all 
ARRA SIG eligible schools 

• State test data for the ARRA SIG schools participating in the study, disaggregated 
by various student populations 

Interi’iews at the state, district, and loccd levels 

CEP conducted in-depth interviews with state and local officials responsible for 
implementing the ARRA SIG program. Local officials included the district official(s) 
responsible for school improvement funds and implementation in the selected schools 
and the principals of the schools that received ARRA SIG funds and the non-recipient 
comparison schools. In some cases, researchers also interviewed teachers at selected 
schools. 

Our choices, including the selection of our interview participants, evolved as our study 
progressed, a common process in qualitative research. As we adapted our theories and 
research questions with each phase of data collection, we also reconsidered our choices of 
interview participants. 

Semi-structured interview protocols were designed with the intent to collect data about 
program implementation, including information about these officials’ general experiences 
and perspectives of successes and/or challenges of the program. The interview protocols 
were informed by prior CEP research on school improvement, as well as our knowledge 
of existing research and federal policies on school improvement. All protocols were also 
reviewed and revised by an external expert in school improvement policy research. 


It is important to note that because we studied three individual states, the data collected 
from these interviews do not represent all states participating in the ARRA SIG program. 
Similarly, each school case within each state case does not represent all ARRA SIG 



schools. Therefore, these interviews are illustrative of trends and examples in each state 
and school studied and are not generalizable beyond those contexts. 

Interviews were conducted both in person and by phone for approximately one hour each. 
The interviews were digitally recorded, transcribed, and analyzed for trends by members 
of the research team. In addition, to ensure the accuracy of the data collected in 
interviews, CEP conducted “member checks,” in which CEP researchers asked 
participants to review transcripts and/or reports resulting from the interviews for factual 
accuracy. 

State and local documents pertaining to ARRA SIG 

CEP reviewed state and local ARRA SIG applications, as well as state and local policy 
statements associated with ARRA SIG, all of which were publicly available. 

U.S. Department of Education existing data set 

CEP researchers used an existing data set published by the U.S. Department of Education 
(Hurlburt et al., 201 1). For each of the states included in the ARRA SIG case studies, 
researchers collected demographic data from the ED data set for all tier 1 and 2 ARRA 
SIG-eligible schools (both recipients and non-recipients) and tier 3 schools in 
improvement under NCLB in school year 2010-11. Variables included the following: 

• Number of students served 

• Grade levels served 

• Locale (rural, suburban, town, urban) 

• Percentages eligible for free or reduced-price lunch 

• Percentages by ethnicity (Asian, Black, Hispanic, Pacific Islander, White) 

• Percentages of English language learners 

• Percentages of special education students 


Analysis 



The research in Maryland, Michigan, and Idaho was modeled as a multiple-case study 
design (Yin, 2003) and allowed us to analyze the data in layers (state, district, and school 
level). Researchers began by analyzing the ARRA SIG applications and any other 
publically available documentation about the program. Through content analysis, these 
state and local documents were reviewed prior to conducting the state and local 
interviews. Content analysis — “a method for analyzing the message characteristics 
systematically... in a way that attempts to be duplicable and comparable” — allowed the 
researchers to generalize about the overall content of a document or a set of documents in 
order to track subtle differences or changes over time (Hesse-Bieber & Leavy, 2006, p. 
47). Although we realize that the background and contextual knowledge gleaned from 
these documents influenced our interview protocols and data analysis, we feel it better 
informed our research questions and interview protocols. 

Once interview protocols were developed for both state- and local-level interviews, 
interviews were conducted by the team of researchers. Often data from the state 
interviews, as well as from our document review and prior research, helped to inform the 
interview questions at the district and school levels. 

Researchers reviewed and coded the interview transcripts independently to extract 
patterns and emerging themes, as well as exceptions to themes within and between the 
data sources. In some instances, we developed concept maps and matrices from these 
data as a tool to assist with the development of emergent themes. This type of 
collaborative analysis allowed researchers to check each other for biases and provided the 
benefit of multiple perspectives. Additionally, it allowed for themes and patterns to 
emerge across cases. 

Throughout the data collection and analysis process, the researchers frequently met in 
person or communicated via telephone and e-mail about the interview findings, coding, 
and content analysis. Ultimately, we wrote summaries of state and local interviews and 
documents that described ARRA SIG implementation, as well as participants' views of 



the successes and challenges of their school improvement efforts. We then used these 
summaries to determine cross-cutting themes. 
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